Optimal Index Policies for Mdps with a Constraint

Many controlled queueing systems possess simple index-type optimal policies, when discounted, average or finite-time cost criteria are considered. This structural results makes the computation of optimal policies relatively simple. Unfortunately, for constrained optimization problems, the index structure of the optimal policies is in general not preserved. As a result, computing optimal policies for the constrained problem appears to be a much more difficult task. We provide a framework under which the solution of the constrained optimization problem uses the same index policies as the non-constrained problem. The method is applicable to the discrete-time IClimov system, which is shown to be equivalent to the open bandit problem.